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Abstract 

This paper presents a dialogical tool for the advancement of learning analytics implementation for student 
retention in Higher Education institutions. The framework was developed as an outcome of a project 
commissioned and funded by the Australian Government’s Office for Learning and Teaching. The project 
took a mixed-method approach including a survey at the institutional level (n = 24), a survey of individual 
teaching staff and other academics with an interest in student retention (n = 353), and a series of 
interviews (n = 23). Following the collection and analysis of these data an initial version of the 
framework was developed and presented at a National Forum attended by 148 colleagues from 43 
different institutions. Participants at the forum were invited to provide commentary on the usefulness and 
composition of the framework which was subsequently updated to reflect this feedback. Ultimately, it is 
envisaged that such a framework might offer institutions an accessible and concise tool to structure and 
systematize discussion about how learning analytics might be implemented for student retention in their 
own context. 


Introduction 

This paper reports on findings from an Australian Government Office for Feaming and Teaching 
commissioned project entitled Fearning Analytics: Assisting Universities with Student Retention. One of 
the primary outputs of this project was a framework based on data collected from institutions and 
academics, using a mixed method approach featuring two different surveys and a series of interviews, 
conducted between July, 2014 and February, 2015. Following the surveys and interviews a draft 
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framework of factors relevant to the institutional implementation of learning analytics for student 
retention purposes was developed. The framework was presented to peers and their feedback elicited at a 
National Forum held in April 2015, based on which the draft framework was refined to arrive at the 
version presented here. 

The primary purpose of this paper is to present a framework of factors relevant to the institutional 
implementation of learning analytics, situate it with other thinking in this area, and discuss how the 
framework might be used in practice. Such a framework may prove a useful tool for institutions engaged 
in decision making and planning around the use of learning analytics for student retention purposes, 
particularly because learning analytics initiatives often require people with a wide variety of expertise to 
work together (e.g. coding and programming, project management, learning and teaching, discipline 
knowledge, statistics, and potentially a range of others). A secondary purpose is to explain the process of 
the framework’s development and explore the context in which this development occurred. Attention to 
this second element is critical as it draws attention to the underpinning thinking linked to previous work 
in the field, and data input highlighting the need for a dialogical tool due to the complexity of learning 
analytics. 

Structurally, the paper begins with a brief literature review in which the two primary concepts 
associated with the framework are explored: student retention and learning analytics. The literature 
review also considers other frameworks and tools related to institutional implementation of learning 
analytics. To conclude the literature review, the research questions and thinking behind these are 
described followed by the research methods. The results section focuses initially on headline findings 
from the data, which leads into presentation of the framework. The paper concludes with some 
propositions about how the framework might be used, discussion of how it fits with other 
conceptualisations of institutional implementation of learning analytics, and a brief overview of future 
directions. 


Literature Review 


Student Retention 

Retention, and more broadly student success, are critical but challenging issues within the higher 
education sector in Australia. Widening participation agendas, which include the desire to have an 
increased number of students and a broader diversity of students, are a high priority (Bradley, Noonan, 
Nugent & Scales, 2008). Key cohorts for improving higher education participation and success are 
Aboriginal and Torres Strait Islander people (Universities Australia, 2013; Behrendt, Larkin, Griew & 
Kelly, 2012), people from regional areas and people of low socio-economic status (Universities Australia, 
2013). 


Responding to this challenge, institutions have been working toward adapting their environments 
and learner support mechanisms. One aspect of this has been the use of technology enabled learning, 
which can provide increased flexibility and personalised learning opportunities. Despite the efforts made 
to date, attrition rates continue to be high, particularly for more diverse learner cohorts with a sector 
average of 17.31% (Koshy, 2014). Additionally, there are concerns around higher attrition rates for 
students studying online (Dekker, Pechenizkiy & Vleeshouwers, 2009; Rivera & Rice, 2002). There is 
some speculation around the reasons for this including differences in student demographics, difficulties 
with effective use of technology and social isolation (Frankola, 2002). 
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The academic and non-academic factors that can influence retention are complex and varied 
(Nelson, Clarke, Stoodley & Creagh, 2014). In a study of business students across six Australian 
universities (n = 7486), Willcoxson and colleagues (2011: 1) reported the following: 

...data strongly indicates that factors related to attrition are generally university-specific and 
reflect both student characteristics and their responses to the specific institutional culture and 
environment. The only attrition triggers which span most universities and most years of study are 
Tack of a clear reason for being at university’ and ‘the feeling of having insufficient ability to 
succeed at university’. 

This suggests that attempts to tackle attrition need to be considered and contextualised to specific 
institutions. It is widely recognised that retention is one facet of student success and that many of the 
actions that we take to improve outcomes for those students who are ‘at risk’ can actually benefit all 
students throughout their learning journey. 

Learning Analytics 

Linked to technology enabled learning and broader digitalisation of interactions (e.g. libraries, 
student services engagements) are a range of opportunities to collect and analyse data with the intention 
of improving the student experience (Gasevic, Dawson, & Siemens, 2015). As Buckingham Shum and 
Ferguson have observed (2012: 3) “the core proposition is that, as unprecedented amounts of digital data 
about learners’ activities and interests become available, there is significant potential to make better use 
of this data to improve learning outcomes”. While there has been much debate around definitions of 
learning analytics and attempts to disaggregate learning analytics from educational analytics, educational 
data mining and academic analytics (Ferguson, 2012; Gasevic et al., 2015; Long & Siemens, 2011; van 
Bameveld, Arnold, & Campbell, 2012), such debates are not seen as useful in the context of this paper. 
Learning analytics is an emerging field and most in the sector are more interested in how data can be used 
rather than the nuances of definitions (Sclater, 2014). For our purposes, learning analytics can be seen as 
the "measurement, collection, analysis and reporting of data about learners and their contexts, for 
purposes of understanding and optimizing learning and the environments in which it occurs" (Long & 
Siemens, 2011). Sclater (2014) notes that such data can be seen to be on a continuum of use value to 
different stakeholders (which is more in line with how institutions perceive it) and can be used by these 
various stakeholders (including students, tutors, educational researchers, unit heads and management) for 
different purposes, depending on their particular focus. 

Mapping Institutional Progress with Learning Analytics 

Irrespective of definitions, the implementation of learning analytics in the sector is at an early 
stage (Colvin, et al., in press; Fiadhi, 2014; Johnson, Adams Becker, Estrada, & Freeman, 2015; Sclater, 
2014; West et al., in press). While a variety of case studies in relation to initiatives is available 
internationally (see Arnold & Pistilli, 2012; Sclater, 2014; Yeado, Haycock, Johnstone, & Chaplot, 2014), 
these tend to be of a relatively small scale, localised or narrow focus. They are useful to demonstrate what 
is possible and to share practice, but do not necessarily move others forward on implementation. Efforts 
to assist in this regard have tended to focus on frameworks for building organisational capacity and/or 
benchmarking more broadly, and related to technology enabled learning, as well as subsequent 
application of these to learning analytics. These have included a variety of foci including measuring 
capability (Marshall and Mitchell, 2002; Marshall, 2006; ACODE, 2014), advocating process (Bichsel, 
2012; Campbell and Oblinger, 2007) and/or identifying domains for attention (Arnold, Lonn, & Pistilli, 
2014; Arnold, Lynch, et al., 2014; Sclater, 2014). Thus, most frameworks generally incorporate each of 
these, but it is useful to highlight some of the main elements. 
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Benchmarking is a critical element of quality control mechanisms and as such the sector often 
looks for ways to assess our development in key areas. Much work has been done on this element related 
to technology enabled learning, including the earlier work of Marshall & Mitchell (2002), and 
subsequently Marshall (2006). Their e-leaming Maturity Model (eMM) is intended to assist institutions to 
deliver e-learning in a sustainable and high quality manner. Building on Chickering and Gamson (1987) 
they initially presented a model based on levels of capability. However, the second version removed the 
concept of levels, acknowledging that it implied a hierarchical structure which was unhelpful in a 
dynamic environment and they moved to one based on dimensions. The overall premise is that there 
needs to be capability across the dimensions which include: learning; development; support; evaluation 
and quality control; institutional planning and management. The concept of process around these is 
overlaid in each dimension. Each dimension then includes a list of statements to prompt reflection and 
rating. At the core, the model provides a way of assessing how an institution rates in relation to each 
dimension, from fully adequate to not adequate. 

Similar to Marshall (2006), Educause’s ECAR model (which is specifically focussed on learning 
analytics) attempts to provide a framework for measuring capability but places more emphasis on the 
domains. The domains included in this model are: process, culture, expertise, investment, 
govemance/infrastructure and data/reporting tools. In a similar vein, The Learning Analytics Readiness 
Instrument (LARI) (Arnold, Lonn, et al., 2014) focusses on 5 domains: ability; data; culture and process; 
governance and infrastructure; and overall readiness perceptions. It is intended for use by institutions to 
develop an understanding of both their strengths and areas for development via rating against statements 
in each domain. 

The use of a series of statements to rate one’s progress is the hallmark of benchmarking 
frameworks. The Australian Council on Open and Distance Education (ACODE) outlines and follows this 
process in the development of their document for benchmarking technology enhanced learning. The 
document (ACODE, 2014) includes a scoping statement, good practice statement and performance 
indicators and measures; it also has room for rationale, evidence, and recommendations for eight 
domains. These domains include governance; planning; information technology (IT) infrastructure; 
application of technology; staff development; staff support; student training and student support. It is 
designed as a tool to be used actively by institutions for self-assessment and comparison. 

Another version that focussed more specifically on learning analytics, and that incoiporates 
stages and domains, is put forward by Norris and Baer (2013). Their model incoiporates the domains of 
technology infrastructure; processes and practices; culture and behaviour; skills and values; and 
leadership. It includes three stages of capacity building from getting started to accelerating progress, and 
lastly to transforming/optimising analytics. 

While most frameworks place some emphasis on process, for some this is a central focus; most 
notably Bichsel (2012) and Campbell and Oblinger (2007). Campbell and Oblinger identify 5 steps, 
which are: capture; report; predict; act and refine. Bichsel notes that while there is a wide degree of 
variation in how analytics might be used, it can be conceptualised as having a common cyclical process 
with the following five stages that loop back as refinement takes place: 

1. Identify a strategic question 

2. Find or collect relevant data to the question 

3. Analyse the data with a view to prediction and insight 

4. Fonn and present the data in understandable and actionable ways 

5. Feedback into the process of addressing the strategic question and identifying new ones 
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Lessons from the Literature 


To conclude this brief overview, it is useful to highlight the themes that arise from the available 
frameworks. They all advocate, to a greater or lesser extent, the need for ongoing refinement and 
improvement via a process of reflection. They draw attention to the fact that development and progress in 
learning analytics is a complex, interdependent and dynamic endeavour, which encompasses a variety of 
stakeholders across the institution. Related to this, a variety of domains need to be attended to, including 
those most commonly noted: technical infrastructure, policy and governance, skills; support and culture. 
Implied within these frameworks, and explicit in the retention literature, is the need to undertake 
development in a way that is appropriate to any given institution. So while it is critical to share examples 
and strategies, there is a need to work more effectively within an institution to make clear progress. 
Working from these key ideas, our project sought to develop a framework for moving forward on the use 
of learning analytics for student retention. 

Project Context and Research Questions 

As learning analytics is at an early stage and considerable variety exists amongst analytics 
methods and products, and the individual contexts in which learning analytics might be deployed, it was 
difficult to hypothesise which themes would emerge most strongly over the course of the study. The 
project team was aware through its own professional networks and involvement in various forums that 
many universities were actively exploring and testing different tools, but that widespread and mature 
usage seemed fairly limited. Consequently, the project adopted a broad and exploratory scope with the 
intention to elicit information from a broad range of contexts. This is reflected by the two main research 
questions: 

1. What factors are relevant or need to be considered where the implementation of learning analytics 
for student retention purposes is concerned? 

2. How do these factors impact on the implementation of learning analytics for student retention 
purposes? 


Method 

Given that this paper is primarily focused on introducing a framework, it is useful to think of the 
method as having two distinct phases. The first is the initial mixed-method (survey and interview) data 
collection that infonned the development of a draft framework. The second phase represents the National 
Forum where the framework was presented to peers for feedback and a process of refinement and 
adjustment that followed. 

Phase 1 


Institution level survey. The institution level survey, which can be viewed here , was conducted 
in July and August 2014, and was directed toward senior academic leaders who could give an overview of 
the institutional strategy around learning analytics. The survey was intended to help build a picture of 
sector progress with learning analytics more broadly and in relation to student retention, as well as data 
infrastructure, human resources, policy development and other elements that relate to learning analytics as 
identified through the literature. It was distributed to Deputy Vice Chancellors (Academic) (DVCA’s) at 
each Australian university via email and promoted via the Universities Australia DVCA forum. Twenty- 
two Australian institutions participated. The sample was extended to New Zealand (NZ) following a 
request from the project reference group which resulted in two NZ institutions participating. The survey 
was built and hosted using the online Qualtrics application and was set to allow anonymous responses. 
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SPSS was used for the quantitative analysis to produce descriptive statistics while manual coding was 
used to analyse qualitative questions. 

Academic level survey. The academic level survey, which can be viewed here, conducted 
between September and November, 2014, was targeted at academic staff (e.g. teachers, student support, 
academic developers etc.), and focused on how they were thinking about and/or using learning analytics 
in the context of student retention. The survey employed a purposive, snowball sampling strategy to 
recruit self-selecting individuals (n = 353), whose responses were anonymous, with invitations circulated 
via existing networks, professional associations and conferences. Analysis involved a range of techniques 
including frequencies, means testing and tests of significance as appropriate to the individual quantitative 
questions and manual coding of themes for qualitative questions. 

Table 1: Frequency distributions of demographic data about academic level survey participants 


Variable 

Category 

Absolute 

Frequency 

Relative 

Frequency 

Location 

Australia 

341 

97% 

(n = 351) 

New Zealand 

10 

3% 

Primary Work Role 

Teaching Students 

188 

53% 

(n = 353) 

Learning Support 

47 

13% 


Management/Administration 

37 

11% 


Other 

32 

9% 


Research 

24 

7% 


Academic Development 

18 

5% 


Student Support 

7 

2% 

LMS at Institution 

Blackboard 

203 

58% 

(n = 353) 

Moodle 

124 

35% 


Brightspace (D2L) 

13 

4% 


Sakai 

2 

1% 


Other 

11 

3% 

Academic Level 

Associate Lecturer/Tutor 

30 

9% 

(n = 351) 

Lecturer 

124 

35% 


Senior Lecturer 

88 

25% 


Associate Professor 

30 

9% 


Professor 

20 

6% 


Other 

59 

17% 

Length of employment in Higher 

Less than 1.5 years 

11 

3% 

Education Sector 

1.5-5 years 

42 

12% 

(n = 345) 

5-10 years 

85 

25% 


10- 20 years 

130 

38% 


More than 20 years 

77 

22% 

Involvement in teaching students 

Teaches students 

276 

78% 

(n = 353) 

Does not teach students 

77 

22% 


Table 1 presents a summary of sample demographics in the form of frequency distributions for a 
range of employment related indicators. While the sample represents a small proportion of the overall 
population of Australian academic staff potentially involved with student retention, it produced strong 
heterogeneity as far as teaching modalities, experience, location and role are concerned. 
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Interviews. A series of semi-structured interviews were conducted between December 2014 and 
February 2015 with self-selecting participants who completed the academic level survey. The purpose of 
the interviews was to explore experiences with some of the key themes and issues uncovered in the 
quantitative elements of the project. Interviews were conducted with 23 people from 15 different 
universities. Participants held a variety of roles (e.g. teacher, educational developer, student support 
officer, librarian, learning analytics project leader, tutor, and learning and teaching leader) and spanned 
different academic levels. Each interview was digitally recorded (audio) and then transcribed verbatim for 
coding and utilisation in the framework development phase. Data from the interviews was thematically 
analysed using manual coding by two researchers. 

Triangulation Process. The two surveys were designed to include question blocks which were 
complementary to provide data on key themes emerging from the literature from different stakeholder 
perspectives. Data from these two sources were brought together to identify where further information 
was needed which informed the interview questions. Thematic analysis of the interviews was then 
brought together with the data from the two surveys to allow for manual comparison of the findings from 
each, identification of trends and key concerns. This was undertaken using a discursive approach by four 
team members. Through this process the headline findings were identified with data collated under each. 
Throughout this iterative process, themes, concerns, tensions and areas of focus were recorded to feed 
into the development of the framework and the associated question sets. 

Phase 2 


The framework and the associated question set were developed by the project team as a graphic 
representation of the key elements identified through phase 1 of the project. The first version was 
presented to the project reference group and their feedback incorporated. In the meantime, the project 
partners utilised the framework to develop case studies and tested it as a tool to inform their own 
institutional thinking. 

The next step was to present the framework at a National Forum for feedback. This took place in 
April 2015, where 148 colleagues from 43 different institutions came together to discuss the findings, 
explore the framework and share examples from their own experiences with learning analytics. The 
project had a budget that allowed each Australian institution to send one fully supported (transport and 
accommodation) representative to the forum. Typically, this person was someone involved in 
implementing learning analytics, or who would be involved if learning analytics were to be implemented 
at their institution (e.g. DVC Academic, Director of Learning and Teaching, and Business Intelligence 
leaders). The rest of the participants made their own way to the forum, though registration was free. This 
resulted in a further group of stakeholders that included teaching staff, educational designers and 
developers and researchers, among others. 

The morning session of the forum focused firstly on broad presentation of the project findings 
followed by a specific introduction to the framework (for details of the program and presentations see 
West et al, 2015a). The afternoon session split participants into two smaller structured workshop groups 
to explore and test the clarity and usefulness of the framework in the context of participants’ own 
institutions and roles. During these workshops participants worked in small groups and members of the 
project team recorded key issues raised. Following the forum, the project team reconvened and amended 
the framework with further advice being provided by the project reference group. 
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Results 


The results that follow are presented in two parts. The first of these is a series of ‘headline’ 
findings that spanned the three Phase 1 data sets (institutional level survey, academic survey and 
interviews). Given space limitations it is impossible to present all of the data tables and graphs but in this 
instance the headline findings are intended to lead into the presentation of the framework, which 
constitutes the second part of the results section. Readers interested in further exploring the data sets and 
findings can find this information on the project website (West et al, 2015a) 

Headline Findings 

Headline finding 1: The sector was at an early stage of development, implementation and 
understanding around learning analytics. Across the institution level survey, academic level survey 
and interviews, the data indicated that whilst pockets of learning analytics use existed in a number of 
institutions, there were many academics with very little awareness or participation in analytics discussions 
and projects. As an example, Figure 1 presents data from the academic level survey that illustrates this 
point. It shows results when participants were asked how often they discussed learning analytics with 
colleagues in different groups, with the darker bars on the right of each series showing less frequent 
discussion. 


210 - 



Teaching Staff Program or School or Learning Students Colleagues in Central L&T Student Support Institutional 

Course Co- Faculty support staff Communities of Group Staff Staff management 

ordinator Management Practice 


Daily L Weekly ■ Fortnightly ■ Monthly ■< Monthly ■ Never 


Figure 1: Frequency of discussion that participants reported with selected colleagues 

The complexity of learning analytics may mean that development will take time and this may 
limit opportunities for academics to be involved and develop confidence with learning analytics in the 
short term. As one interviewee involved in learning analytics implementation stated: “1 guess 1 am 
looking at where we might end up focusing on the spectrum of learning analytics and 1 don’t really have 
an answer for that just yet”. This is one quote that was indicative of many similar responses. 

Headline finding 2: There was a degree of cautiousness reflecting an emerging recognition 
of the amount of thinking that needs to go into how learning analytics can be applied. Following on 
from the first headline finding, it became clear that perceptions differed on what learning analytics 
implementation might look like, as the following interview quote suggests: 

At our university, we’re not too far advanced. We don’t have that kind of system integration yet, 
that allows us to do that. Before we go headlong into assuming that it is all about system 
integration and spending huge amounts of money on trying to make systems talk to each other 
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that don’t want to, that if we drive the conversation on the basis of understanding what are the 
key points of data, it might be that those systems don’t actually necessarily need to talk to each 
other, but we might be able to aggregate that information at another level. 

This is where variation in leadership and strategic thinking can impact on what gets implemented 
and when. In some institutions there was a clear expression by those setting the direction that getting data 
into integrated systems, such as a data warehouses, would create the flexibility to address questions as 
they arise - one did not necessarily need to know what those questions were at the time of building the 
data warehouse. In contrast, other participants’ views were more indicative of the quote above, which 
reflects a problem by problem approach in which implementation is driven by clearly identified needs and 
problems. 

Headline finding 3: There was great variability across the sector - in relation to 
preparedness, how institutions are thinking about and implementing learning analytics due to 
context. The first two headline findings illustrate the complexity and broad scope of learning analytics 
and how this complexity posed challenges for participants in the study. The third headline finding follows 
logically but speaks to a different kind of complexity, which is the significant variation that can occur 
across institutional contexts and environments. This can take many forms, for example, variation in 
organisational culture as articulated by one interviewee: 

1 think the real learning for me so far has been the need for the cultural environment at the 
institution and the associated drivers and leadership. The data has to have a certain environment 
or a certain context before it has any meaning whatsoever. That way you can clearly mark what 
that is and you can find a platform for it to have a sort of life within the environment. 

Another commonly expressed example of variation in institutional context related to the student 
cohort. One participant described the student cohort at their university as “fairly competitive”, another 
spoke of a large population of “low socio-economic” students, whilst another again spoke of their 
university being very “research focused”. Taken together, these examples point to the importance of 
institutional context in shaping how learning analytics might be implemented and the framework unpacks 
this further. 

Headline finding 4: Participants identified tensions and questions around the extent to 
which learning analytics can drive actions and behaviours or take the functions of people. Perhaps 
not surprisingly, given the Figure 1 data that illustrated the limited amount of analytics focused discussion 
taking place involving faculty, there was some uncertainty about how students would experience learning 
analytics, particularly automated communication. This tension was captured by one participant, who 
observed: 

Even though you are using automated systems to communicate and provide feedback you still 
have to make sure that it is personalised and meaningful for students and that takes time and 
consideration.. .In the system you can use this code so you don’t have to write names every time 
and you don’t have to use hello everyone all the time. This gets more attention from students and 
encourages students to be engaged. 

Another participant offered the view that there would still be a major role for direct person-to- 
person communication with students. 

Statistics can be helpful, but it can also be useful to talk to students directly and see what they 
actually think they need themselves. From all of this we can work on programs, strategies and 
supports that might help them stick around a bit longer. 
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Whilst these two quotes express a consciousness of student needs they also potentially allude to a 
broader fear that one’s job or work role might be impacted on, or even superseded, by computers. 

Headline finding 5: There were substantial gaps between academics’ needs and aspirations 
and what is being provided, what is available, and what is being developed. One of the challenges the 
sector faces in widely leveraging the potential of learning analytics is how to distribute knowledge and 
awareness of what can be achieved through learning analytics. As the previous quotes illustrated there are 
both real and potential fears around the implications of learning analytics for academic staff, particularly 
where academics are not well informed about what learning analytics are, how their institution plans to 
use them, and how they are expected to use them. The data in Figure 2 reinforces this point. Taken from 
the academic level survey, it shows that a large proportion of participants rated their institution as ‘poor’ 
or ‘very poor’ at meeting their needs in relation to some basic learning analytics provisions. Further 
illustrating the uncertainty felt by participants is that a relatively high proportion were unsure how they 
would rate their institution - n for ‘not sure’ varied between 85 and 104 across the six variables. 
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90 

60 

30 
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Provision of 
information about 
how learning 
analytics use will 
affect me (n = 197) 


Opportunities to 
provide feedback 
about learning 
analytics 

implementation (n 
= 195) 


Ease of 

visualisation and 
interpretation of 
data (n= 190) 


Provision of 
information about 
how learning 
analytics is being 
used (n = 204) 


Relevance and 
comprehensiveness 
of data that I can 
access (n= 192) 


Ease of learning 
analytics data 
access (n= 187) 


Provision of 
professional 
development about 
learning analytics 
(n= 199) 


Poor or Very Poor BFair ■ Good or Very Good 


Figure 2: Participants rating of their institution at meeting their needs around selected learning 
analytic provisions (excludes ‘not sure’ responses to show trends visually) 

The messages in this data are consistent with data provided by a number of interviewees as well 
as the institution level survey, which suggested many institutions were currently engaged in localised 
pilot projects and/or institution level systems development, which for various reasons may not be well 
publicised to the wider staff cohort in both cases. This leads to the final headline finding. 

Headline finding 6: There was tension between business needs, wants and limitations (e.g. 
costs) versus academic staff needs and wants (e.g. academic freedom, innovation in learning and 
teaching). A recurring theme across the two surveys and interviews was that the roles of individual staff 
and work groups are a key factor in determining the types of learning analytics applications they might be 
interested in, and to some extent this is where the ambiguity around educational and learning analytics has 
impacts. There are two parts to this. 

The first is that a number of participants expressed a desire for analytics to help inform their 
work, but there was also an expressed desire for academic freedom i.e. not being told how to teach, what 
forms grades had to be entered, or when to commence assessment tasks, for example. This can be 
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problematic when particular processes need to be followed in order that data is cleanly and consistently 
recorded and thus usable. The second part relates to the beneficiaries of learning analytics. There are 
questions about the data that might be used for learning analytics, especially who owns it and how it can 
and should be used. When participants were asked about the ethical principles that should guide learning 
analytics, a number talked about the highest priority being student benefit. Others went as far as saying 
that the institution should not be using learning analytics for its own benefit. 

Linking the Data and the Literature: A Summary of Lessons 

These headline findings and a more granular thematic analysis of survey and interview responses 
(West et al, 2015) led the project team to consider how the various frameworks/models and benchmarking 
documents assist in implementation and address the issues described by participants. Key lessons from 
this reflection included: 

• Despite a range of examples of learning analytics initiatives being available, people in the sector 
are struggling with institution level implementation of learning analytics, especially in terms of 
aligning policies, systems, people and stakeholder needs. 

• Institutional context will change the shape of development and pathways for implementation 

• A reflective, non-linear model is essential 

• The domains outlined in the literature are essential but not sufficient, particularly in trying to 
address student success 

• The models/frameworks are more broadly about technology enabled learning and/or learning 
analytics but are not focussed on the issue of student success 

• Currently, there is a lack of clear alignment between the work of researchers, institutional 
learning analytics leaders and academic staff, particularly in terms of working together on 
problems of agreed strategic importance. The data in Figure 1 showed that academics in this 
sample were engaging in a relatively limited amount of dialogue with other colleagues, making 
strengthening dialogue a logical starting point. 

Introducing the Let’s Talk Learning Analytics Framework 

With these assumptions in mind, the Let’s Talk Learning Analytics Framework was developed 
(and refined as per the methodology above). It consists of two primary documents: 

1. A one-page high level summary of the framework (Figure 3), which consists of six key domains. 
The six domains, shown in green, are the areas an institution needs to consider to make progress 
on learning analytics for retention. They are interconnected and shaped by the primary domain, 
the institutional context. Continual reflection and adjustment are critical to the process and will 
require input from across the institution. 

2. A set of discussion questions (West et al, 2015c) to support the use of the framework, organised 
around the key domains. The purpose of the discussion questions is to provide a series of prompts 
to guide a contextualised discussion and is based on the premise that dialogue across an 
institution is critical. 

The set of questions which accompany the framework are essential but, for this paper, will be 
split and presented under each of the domains to help illustrate the relevance of those domains. In practice 
it would be expected that learning analytics leaders would use the questions and framework documents 
side-by-side in a meeting, workshop, or any other environment centered on dialogue, or a combination of 
these in line with the yellow section in Figure 3. The questions are not statements related to stages or 
levels but rather, prompts, to engage in discussion and reflection to ensure that key elements are 
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articulated, clarified or marked for development. The framework is also seen as dynamic and non-linear 
with continual reflection and adjustment, based on the idea of transitioning to learning analytics use. The 
term ‘transitional’ is used to reflect that learning analytics is an emerging field and many institutions will 
be transitioning to an environment that, for the first time, broadly incorporates learning analytics (e.g. into 
institutional policies and plans), as well as transitioning to an environment where learning analytics is 
specifically applied to issues like student retention. As such there is no implied point for starting the 
process beyond the institutional context. There is an expectation that institutions will engage in a process 
of praxis (reflection, action, reflection). 


Locati< 


INSTITUTIONAL CONTEXT 

Institutional context provides the parameters for what is feasible/appropriate and includes: 

Student demographics and characteristics Staff demographics and characteristics Size and structure Strategic positioning of the institution 

SYSTEMATIC DISCUSSION OF THE SIX DOMAINS - GUIDED BY THE DISCUSSION QUESTIONS RESOURCE 


This discussion interface recognises that sucessfully coping with the complexity of learning analytics requires a collaborative, systematic approach because: 


Expertise relating to the six different domains is distributed across institutions 


Learning analytics projects typically exert broad impact across an institution 


TRANSITIONAL INSTITUTIONAL LEARNING ANALYTICS 
ELEMENTS INFRASTRUCTURE 


Transitional institutional 
elements provide the 
parameters for the 
implementation of learning 
analytics. They include: 

Culture 

Positioning of learning 
analytics within the 
institution 

Level of sponsorship 

Governance arrangements 

Alignment with institutional 
strategy 


Learning analytics 
infrastructure is concerned 
with three main factors: system 
reliability, system 
sophistication and relevant 
expertise. This includes: 

Digital availability and 
integrity of data 

Integration, continuity and 
availability of data systems 

Technical, pedagogical, 
statistics, and project 
management expertise 

Data stewardship 


TRANSITIONAL RETENTION 
ELEMENTS 

Transitional retention elements 
provide the parameters to 
enable more effective 
deployment of learning 
analytics for retention 
purposes. These include: 

Retention planning 

Retention strategy and 
implementation 

Governance arrangements 
related to student retention 


LEARNING ANALYTICS 
FOR RETENTION 

Learning analytics for 
retention factors are focussed 
on the use of learning analytics 
for retention and include: 

Educational and business 
questions from various 
stakeholders 

The ability of the system to 
address the questions 

Accessibility, ease of use of 
system, tools and reports for 
various stakeholders 

Consideration and resolution 
of the ethical issues which 
may arise from the 
implementation and use of 
learning analytics 


INTERVENTION AND 
REFLECTION 

Intervention and reflection 
are critical to improving 
retention. Consideration 
needs to be given to: 

Training, support and time 
for staff and students to use 
systems, interpret data and 
reports and act on them 

Endorsed processes around 
actions or interventions 
arising from the data 

Modification of relevant 
elements, systems, factors 
and interventions 


Sustainability 


Policy and procedures 


Figure 3: Let’s Talk Learning Analytics Framework 


Domain 1: Institutional Context. The central message from the data was that institutional 
context is the guiding feature of analytics implementation. It sets the scene for all of the decisions that 
will follow and as such is seen as foundational and therefore positioned in the framework across the top to 
highlight the fact that it will pervade all other domains. The questions for consideration in this domain 
include: 


• What are the implications of the jurisdictional location of your institution (e.g. privacy laws)? 

• What is the size and structure of your institution? 

• What is the strategic positioning of your institution (e.g. partnerships, international, and areas of 
focus)? 

• What are the student demographics and characteristics? 

• What are the staff demographics and characteristics? 
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Domain 2: Transitional Institutional Elements. Transitional elements in this framework are 
used as a term to both narrow the focus and to suggest that the elements that sit in these domains are a 
lens by which to view other domains and elements. They will inevitably shape the development of 
learning analytics development in an institution. For example, in institutional transitions, the executive 
commitment and position is critical to progress as it will impact on investment, practices and culture (see 
Arnold et al, 2014 and Norris & Baer, 2012). Key questions for discussion in this domain include: 

• How does the culture at your institution both promote and challenge the advancement of learning 
analytics? 

• What is the strategic positioning of learning analytics within your institution? 

• To what degree is there executive sponsorship (academic and IT)? 

• Do you have a specific learning analytics plan and if so, how is it aligned with your institutional 
budget and academic plan? 

• To what extent is there congruence between the vision, strategy and values of the institution that 
relate to learning analytics? 

• How clearly is the role of learning analytics in your institution identified and communicated? 

• How sustainable is the approach to learning analytics at your institution? 

Domain 3: Learning Analytics Infrastructure. The Learning Analytics Infrastructure domain is 
a representation of the importance of data systems and infrastructure to the use of learning analytics. 
There is no doubt that institutions have been collecting and storing significant amounts of data for years 
(Educause, 2010; Heathcote & Dawson, 2005). However, these have not necessarily been accessible in 
the right format or able to be used effectively. Project participants cited numerous examples of issues 
around data and systems where the interface is not user-friendly, the data is incomplete, data integration is 
inadequate, or the time taken to obtain a report or visualisation is too long. 

This domain includes three key questions which are then subsequently broken down into more 
specific questions under the headings of digital availability of data, data integrity, systems continuity, 
availability and reliability, integration of data systems, expertise, stewardship and data policy and 
procedures. For our purposes only the three key questions are presented here (the fuller set area available 
on the project website (West, 2015c). 

• Are your systems reliable? 

• Are they technically sophisticated enough? 

• Is there enough expertise in the institution? 

Domain 4: Transitional Retention Elements. As this framework is specifically focussed on the 
use of learning analytics for student retention, this domain draws attention to that lens and transitions us 
from thinking about all the things learning analytics might be used for, to how it can be used to support 
student success. Key questions in this transition domain include: 

• Do you have an institutional retention plan or strategy? 

• What are the governance arrangements that support the retention plan or strategy? 

• How are learning analytics positioned within the retention plans and associated governance 
arrangements? 

It is important to reiterate that this framework was developed for a project with a specific focus on student 
retention. This explains the focus of Domains 4 and 5. It is possible to adapt these domains to other key 
foci by changing the focus of Domains 4 and 5, whilst keeping the other domains as they are. 
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Domain 5: Learning Analytics for Retention. This domain highlights key questions that prompt 
thinking around the more specific area of learning analytics for retention (rather than business 
intelligence, action analytics etc.) and so should be considered in that light. 

• What business and educational questions do your stakeholders have, and how are they prioritised? 

• To what extent can your system address those questions? 

• Where gaps are identified, what resources are available for follow up? 

• What ethical issues have been identified and what is the resolution? 

Domain 6: Intervention and Reflection. Data gathered throughout the project demonstrated that 
there is a wide range of learning analytics activities taking place throughout the sector. However, many of 
these are small scale or pilot projects are not necessarily centrally led and take place in various locations 
with different intended outcomes. Yet, projects with no action will have little impact on student success 
and they thus need to be connected to some sort of intervention (whether the data informs an action 
(utilising people to call students) or be part of the action (e.g. automated messages to students or 
connections to personalised learning). Intervention at this stage also includes things like training programs 
for staff and students, which was clearly identified as a gap in the data. 

It was also clear that while reflection is necessary in an ongoing manner, it needed to be more 
explicit in this area to ensure that lessons from learning analytics projects are fed back in to other work. A 
clear example of this might be that a learning analytics project demonstrates that a particular strategy in 
the retention plan is not effective. This should prompt investigation and subsequent revision of the 
retention plan. Key questions in this domain include: 

• How are potential interventions developed, managed and enabled? 

• What training is provided, and what training is required for staff and students? 

• What time and support is provided, and what time and support are required, for staff and 

students? 

• How are the interventions evaluated and improved? 

• How is the whole system evaluated and improved? 

Discussion 

While this framework builds on prior work (Arnold, Lonn, et al., 2014; Bischsel, 2012; Norris & 
Baer, 2012), it has some key differences. These include an emphasis on dialogue, a focus on the use of 
learning analytics for student retention and success, and placing an explicit focus on both the foundation 
of context and the action/intervention required. It attempts to draw attention to the inter-related elements 
that will take learning analytics in different, but equally useful directions. 

Clearly the level of maturity and capacity that institutions have when it comes to learning 
analytics varies and though there has been much research interest in learning analytics, this is not 
necessarily translating into coherent, widely understood plans and strategies within institutions. The flow- 
on effect of this is that academic staff may be anxious about falling behind with a new set of tools (i.e. 
learning analytics) that they keep hearing about. Additionally, they may not be aware that their 
knowledge and skills could be harnessed in relation to institutional learning analytics work. In this way, 
the framework is seen as a key tool for creating constructive dialogue around the potentially daunting 
changes that learning analytics bring. 
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The Role of Dialogue 


The emphasis on dialogue hinges on several factors that were very evident in both the literature 
and the data. First, that learning analytics is complex and in a nascent stage of development, which means 
that we are still exploring the potential that it affords. In order to explore and leverage the potential, we 
need to have the involvement of people from multiple disciplines and backgrounds (Balacheff & Lund, 
2013; Siemens, Dawson & Lynch, 2013) (but particularly IT and pedagogy), and points of view (such as 
teaching academics, head of schools, senior executive and students) at every stage. This includes 
planning, development, deployment, and feedback. Yet the data suggested that such knowledge and input 
was largely occurring in fairly narrow silos or select groups, and teaching academics in particular were 
mostly unaware of the initiatives taking place in their own institutions. As noted in Figure 1, most 
academics in the survey very rarely had discussions with anyone around learning analytics. This is not 
particularly surprising given that many initiatives were small pilot projects of limited scope. Additionally, 
the complexity of learning analytics initiatives from a technical view has placed it largely in the realm of 
IT departments or learning analytics specialists. Breaking down the potential silos is critical and dialogue 
is one way to do this. 

Furthermore, many people (even at higher levels) in the sector have had limited practical 
exposure to learning analytics and therefore do not always recognise tangible implications that impact on 
implementation. Finally, learning analytics implementation necessitates collaboration between people 
with vastly different roles and expertise. For example, the use of even basic reporting for functions from 
software packages which integrate the LMS and SIS, need a level of standardisation in practice to be 
effective. Items such as the grade centre need to be set up in an appropriate way in order to display some 
of the reports. Conversations about these factors and which reports would be most useful from the 
perspective of the teaching academic, could help in a range of ways including to prioritise work in 
analytics implementation; aid in a level of standardised practice by teaching academics; and, link to 
support mechanisms for staff and students. 

Bichsel (Bischsel, 2012) recommends that institutions use a maturity index (specifically the 
Educause maturity index) to gauge analytics progression, set goals and measure progress. While this type 
of benchmarking is useful, the idea of plotting progression on a scale is generally of interest to particular 
stakeholders. Additionally, it is not designed to be an implementation tool or guide but it is about 
measuring progress instead. In contrast the framework presented here is not about benchmarking but 
about providing a set of discussion questions to be actively used to move implementation (or at least 
dialogue on analytics) forward. This is very much in line with the suggestions of Bichsel (2012) who 
recommends attention to planning, process and partnership across various stakeholders. 

Analytics for Student Retention and Success 

As noted at the outset of this paper, the area of student retention and success is a high priority in 
the sector. This focus has flowed through to the use of learning analytics being harnessed to assist in this 
area (for example see (Arnold & Pistilli, 2012; Sclater, 2014; Yeado et al., 2014). Yet frameworks and 
benchmarking tools on analytics are about analytics more broadly, which, while useful, does not prompt 
stakeholders to think about the complex area of student retention nor to frame the work around the 
broader context of student success. While perhaps there is an assumption that that we would be using 
learning analytics to improve the student journey, there is value in making it more explicit. 

While the authors acknowledge and support the idea that learning analytics can be used for a wide 
range of purposes, central to this is the idea of student retention and success. This project was funded to 
consider learning analytics in light of retention but it was very apparent that thinking in the sector has 
moved to the broader construct of student success with retention being a subset of that. In many ways a 


Online Learning - Volume 20 Issue 2 - June 2016 


44 


clear focus on the student journey may help to move the learning analytics journey forward. As Siemens 
(2012: 4) observes: “A transition in analytics from a technical orientation to one that emphasizes sense¬ 
making, decision-making, and action is required to increase interest among educators and administrators. 
This transition in no way minimizes the technical roots of analytics; instead, it recasts the discussion to 
target problems that are relevant to practitioners instead of researchers”. It is perhaps the student journey 
which may be able to provide a point of reference for engagement of a more diverse set of stakeholders. 

The work in the area of student retention can also provide some guidance in terms of the types of 
initiatives which may be useful, and this will flow through to the types of data that would be collected, the 
types of reports required, who should have access to those reports and the subsequent actions that might 
take place. By way of example, Transition Pedagogy (Kift, Nelson & Clarke, 2010; Kift, 2009; Kift & 
Nelson, 2005) focuses on the concepts of transition, diversity, design, engagement, assessment and 
evaluation as being critical to first year retention. If an institution was using such a framework it would 
be much clearer on how to focus learning analytics development. Additionally, some of the concepts 
within Transition Pedagogy are notoriously difficult to measure and learning analytics may offer some 
opportunities to revisit such work to both test and expand the model. 

Context 

Context is critical. This basic statement was a central theme of the data in terms of the initiatives 
occurring, what was possible, the resources and infrastructure available and the priorities set to name just 
a few. In this sense, case studies and examples of what has worked in other places are useful but not 
sufficient. A good example of context within a domain is the strategic positioning of learning analytics 
within an executive portfolio. In some institutions this sits within an academic portfolio and in others in 
the IT portfolio. Such position in one or the other is both a reflection of how the institution views learning 
analytics and its purpose, as well as a driving force for future direction. Starting with the context will 
ensure that the initiatives that take place are appropriate to that environment or at least considered in this 
light. 


Leveraging analytics is a dynamic and interactive process. There are a range of implementation 
decisions which need to be made along the way. There is the potential for these to be driven by a narrow 
focus rather than an institutional one, potentially leading to a mismatch between the decision and the 
direction of the institution, or, if present, the retention plan. For example, the decision to turn on a student 
facing dashboard is deceptively simple but may require deeper consideration in some contexts. 
Consideration may need to be given to the support that is provided around this, the nature of the report or 
dashboard, and the type of follow-up that takes place. Additionally, such dashboards should be built in 
relation to key pedagogic or retention questions. It is essential that each one of these decisions is in line 
with the overall plan and approach, and seen in context. The point is that there is no simple right or wrong 
but rather decisions that are built in line with institutional elements. 

Finally, the decision to pursue learning analytics is costly in terms of time, expertise and money. 
If the decision and subsequent implementation is done in a systematic way and includes various 
stakeholders and expertise in development, it is more likely to have an impact. A number of participants 
emphasised that inadequate communication, where learning analytics implementation is concerned, is 
liable to cost time and money. Therefore, from an efficiency point of view the use of a dialogical 
framework can prompt the communication in a focussed way and avoid unnecessary delays and align 
processes for outcomes. 
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Limitations 


Prior to the concluding remarks some potential limitations are acknowledged: 

• Learning analytics is rapidly developing area so it is likely that ongoing data collection is required 
to gather updated perceptions, experiences and aspiration from academics and institutions as they 
get more experience with learning analytics. To address this limitation, the project team plans to 
conduct follow up data collection in the coming period. 

• The data used to develop the framework was from Australia and New Zealand, so readers will 
need to consider how this might impact on the applicability of the framework in their location, or 
other locations. However, because it is a discussion framework the researchers are interested in 
hearing feedback from peers in other locations and this will potentially assist in further 
refinement. 

• The process of developing and refining the framework via data and various forums was highly 
iterative and describing the nuances of that method for the purposes of replicability is a challenge. 
Questions can be directed to the researchers. 

• In terms of this specific paper it must be acknowledged that space restrictions also limited the 
amount of raw data that could be presented. 

Conclusion 

This paper has presented findings from a mixed-method study of Higher Education institutions 
and academics from Australia and New Zealand. Whilst this project produced a range of data on the 
perceptions, experiences and aspirations around learning analytics, one of the major aspirations of the 
project was to translate this data into a framework to promote institutional implementation bolstered by 
robust and systematic discussion. This has taken the form of the Let’s Talk Learning Analytics 
Framework, and its presentation has been the primary focus of this paper. The framework can be thought 
of as comprising two complimentary tools. The first is the map of domains and factors relevant to 
institutional implementation of learning analytics and the second is a series of discussion questions that 
act as prompts within each of the domains. 

Overall the framework is designed to foster a collegial approach to the implementation of 
learning analytics which is based on providing prompts for dialogue amongst key stakeholders. In this 
way it compliments existing frameworks and models which are more oriented toward measuring progress. 
Ultimately, it is hoped that this will assist in positioning learning analytics in a shared space which 
equally values pedagogical and technical expertise. Most importantly, it suggests that a focus on the 
student journey and the associated operational plans of an institution will be more likely to connect 
analytics to action for the benefits of students. 

In the same way that it takes a community to raise a child, it takes an institutional community to 
truly make the most out of learning analytics. 
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